{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Electricity Demand in Victoria, Australia \n",
    "\n",
    "In this notebook we will prepare and store the electricity demand dataset found [here](https://github.com/tidyverts/tsibbledata/tree/master/data-raw/vic_elec/VIC2015).\n",
    "\n",
    "**Citation:**\n",
    "\n",
    "Godahewa, Rakshitha, Bergmeir, Christoph, Webb, Geoff, Hyndman, Rob, & Montero-Manso, Pablo. (2021). Australian Electricity Demand Dataset (Version 1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.4659727\n",
    "\n",
    "**Description of data:**\n",
    "\n",
    "A description of the data can be found [here](https://rdrr.io/cran/tsibbledata/man/vic_elec.html). The data contains electricity demand in Victoria, Australia, at 30 minute intervals over a period of 12 years, from 2002 to early 2015. There is also the temperature in Melbourne at 30 minute intervals and public holiday dates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download the data via the URL below and pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Electricity demand.\n",
    "url = \"https://raw.githubusercontent.com/tidyverts/tsibbledata/master/data-raw/vic_elec/VIC2015/demand.csv\"\n",
    "demand = pd.read_csv(url)\n",
    "\n",
    "# Temperature of Melbourne (BOM site 086071).\n",
    "url = \"https://raw.githubusercontent.com/tidyverts/tsibbledata/master/data-raw/vic_elec/VIC2015/temperature.csv\"\n",
    "temp = pd.read_csv(url)\n",
    "df = demand.merge(temp, on=[\"Date\", \"Period\"], how=\"left\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Public holidays in Australia\n",
    "url = \"https://raw.githubusercontent.com/tidyverts/tsibbledata/master/data-raw/vic_elec/VIC2015/holidays.txt\"\n",
    "holidays = pd.read_csv(url, header=None, parse_dates=[0], dayfirst=True)\n",
    "holidays.columns = [\"date\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Process and save the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will only use the `OperationLessIndustrial` demand. So let's drop `Industrial`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop(columns=[\"Industrial\"], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's extract the date and date-time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the integer Date to an actual date with datetime type\n",
    "df[\"date\"] = df[\"Date\"].apply(\n",
    "    lambda x: pd.Timestamp(\"1899-12-30\") + pd.Timedelta(x, unit=\"days\")\n",
    ")\n",
    "\n",
    "# Create a timestamp from the integer Period representing 30 minute intervals\n",
    "df[\"date_time\"] = df[\"date\"] + pd.to_timedelta((df[\"Period\"] - 1) * 30, unit=\"m\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop the null rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create holidays column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "holidays[\"is_holiday\"] = 1\n",
    "df = df.merge(holidays, on=[\"date\"], how=\"left\")\n",
    "df[\"is_holiday\"] = df[\"is_holiday\"].fillna(0).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now just use the timestamp and the electricity demand and resample to hourly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>demand</th>\n",
       "      <th>temperature</th>\n",
       "      <th>is_holiday</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date_time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2002-01-01 00:00:00</th>\n",
       "      <td>6919.366092</td>\n",
       "      <td>32.6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002-01-01 01:00:00</th>\n",
       "      <td>7165.974188</td>\n",
       "      <td>32.6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002-01-01 02:00:00</th>\n",
       "      <td>6406.542994</td>\n",
       "      <td>32.6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002-01-01 03:00:00</th>\n",
       "      <td>5815.537828</td>\n",
       "      <td>32.6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002-01-01 04:00:00</th>\n",
       "      <td>5497.732922</td>\n",
       "      <td>32.6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          demand  temperature  is_holiday\n",
       "date_time                                                \n",
       "2002-01-01 00:00:00  6919.366092         32.6           1\n",
       "2002-01-01 01:00:00  7165.974188         32.6           1\n",
       "2002-01-01 02:00:00  6406.542994         32.6           1\n",
       "2002-01-01 03:00:00  5815.537828         32.6           1\n",
       "2002-01-01 04:00:00  5497.732922         32.6           1"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Rename columns\n",
    "timeseries = df[[\"date_time\", \"OperationalLessIndustrial\", \"Temp\", \"is_holiday\"]]\n",
    "\n",
    "timeseries.columns = [\"date_time\", \"demand\", \"temperature\", \"is_holiday\"]\n",
    "\n",
    "# Resample to hourly\n",
    "timeseries = (\n",
    "    timeseries.set_index(\"date_time\")\n",
    "    .resample(\"H\")\n",
    "    .agg(\n",
    "        {\n",
    "            \"demand\": \"sum\",\n",
    "            \"temperature\": \"mean\",\n",
    "            \"is_holiday\": np.min,\n",
    "        }\n",
    "    )\n",
    ")\n",
    "timeseries.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the timeseries in the datasets folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "timeseries.to_csv(\"../Datasets/victoria_electricity_demand.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
